DRAFT: Tiling Stage Fixes and Improvements #65

zthorson · 2021-08-03T18:11:25Z

Description

Fixes to numerous small bugs and odd behaviors discovered when trying to use the tiling stage for the first time in a while.

There are also some improvements to make the output from the tiler more compatible with different tile viewing tools, including those that doesn't implement the TMS standard.

Reason

There are a few locations where the code crashes when creating higher zoom tiles
Higher zoom tiles have their valid data overwritten with blank data both in rgb and elevation_angle files causing zooms above max to be missing a lot of data
There also appears to be a small shift in the x axis relative to the full mosaic at different zoom levels. This may just be pixel rounding, or may be something else. It is TBD.

Method / Design

Perform a check in cv_grid_map that the merged tiles will fit inside the pixel space of the roi's
Attempt to handle NaNs in floating point matrices better to avoid ambiguities
Update blend routines to match those used in mosaicing. The older method had translucent pixels around the stitch location.
Add a use_tms flag to the tiling.yaml file to allow using TMS, or Google/OSM standard tile names in the cache. This primiarily inverts the y axis. See here: https://www.maptiler.com/google-maps-coordinates-tile-bounds-projection/

Testing

Compiled and run on:

Ubuntu 20.04 - Desktop
Custom OpenREALM wrapper (non-ROS)
Fed results of ortho stage to both mosaic and tiling for comparisons

Other Notes

- Comparisons using OpenCV and NaN numbers can have undetermined results due to some shortcuts OpenCV takes in performing comparison operations - This can results in odd behavior where only some of the NaNs are included

laxnpander · 2021-08-04T12:41:28Z

@zthorson One thing about the shift: Did you realize that the tileing is using EPSG:3857? This is slightly different from UTM coordinates and could be the root of the shift depending on the visualization that you use.

Edit: As a frame of reference I use this file for my dataset to visualize the tiles. It must be copied into the folder where the different zoom levels are.

<!DOCTYPE html>
<html>
  <head>
    <title>ortho</title>
    <meta http-equiv="imagetoolbar" content="no">
    <meta http-equiv="Cache-control" content="no-cache">
    <!--meta http-equiv="refresh" content="0.2"-->
    <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
    <meta name="apple-mobile-web-app-capable" content="yes">
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.3.1/leaflet.css">
    <link rel="stylesheet" href="https://cdn.rawgit.com/ardhi/Leaflet.MousePosition/master/src/L.Control.MousePosition.css">
    <script src="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.3.1/leaflet.js" type="text/javascript"></script>
    <script src="https://cdn.rawgit.com/ardhi/Leaflet.MousePosition/master/src/L.Control.MousePosition.js" type="text/javascript"></script>
    <style>
      body { margin:0; padding:0; width:100%; height:100%; background: #ffffff; }
      #map { position:absolute; top:0; bottom:0; width:100%; z-index: 1; }
      #slider{ position: absolute; top: 10px; right: 10px; z-index: 5; }
    </style>
  </head>
<body>
  <div id="map"></div>
  <!--input id="slider" type="range" min="0" max="1" step="0.1" value="1" oninput="layer_rgb.setOpacity(this.value)"-->
  <script type="text/javascript">
    function sleep(ms) {
      return new Promise(resolve => setTimeout(resolve, ms));
    }
    var generateRandInt = function() {
      return Math.floor( Math.random() * 200000 ) + 1;
    };

    var mapExtent = [10.22556861, 52.40028879, 10.23224548, 52.40437500];
    var mapMinZoom = 15;
    var mapMaxZoom = 20;
    var bounds = new L.LatLngBounds(
      new L.LatLng(mapExtent[1], mapExtent[0]),
      new L.LatLng(mapExtent[3], mapExtent[2]));
    var map = L.map('map').fitBounds(bounds);
    L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png').addTo(map);
    var options = {
      randint: generateRandInt,
      minZoom: mapMinZoom,
      maxZoom: mapMaxZoom,
      opacity: 1.0,
      attribution: 'Rendered with OpenREALM>',
      tms: true
    };

    var layers = [L.tileLayer('{z}/{x}/{y}.png?{randint}', options).addTo(map)];
    setInterval(function() {
      layers.push(L.tileLayer('{z}/{x}/{y}.png?{randint}', options).addTo(map));
      if (layers.length >= 5) {
        layer_deleted = layers.shift();
        map.removeLayer(layer_deleted);
      }
      console.log(layers.length)
    }, 200);
  </script>
</body>
</html>

zthorson · 2021-08-05T21:36:29Z

@laxnpander I was using QGIS to visualize the tiles, but it seems like that file may work better, as I won't have to deal with the caching that QGIS uses.

I'm noticing a nice memory footprint improvement using the tiling stage instead of the mosaicing one that I'm hoping to test on a device soon as well.

- Merge, even with the previous patches, appears to improperly detect which tiles to overwrite when creating zoomed out tiles. This results in empty mosaic/elevation data overwriting good data in some tiles. - The real fix would be to figure out what is going on with merge, but blend() is a good workaround

laxnpander · 2021-08-06T07:48:41Z

@zthorson In theory, memory required should be more or less constant as all unnecessary data is being written to the disk and removed from the RAM and only relevant tiles dynamically loaded. However, this of course comes with more read/write operations and possibly a significantly higher CPU load. So you will have to check whether this works for you or not.

In case you hit a performance sink somewhere, I have two things in mind where you might want to make adjustments:

The tile cache has a rather simple caching strategy. I basically check what ROI was requested last time and I try to predict what tiles might be new per zoom level:

OpenREALM/modules/realm_ortho/src/tile_cache.cpp

Line 372 in 465ecc8

void TileCache::updatePrediction(int zoom_level, const cv::Rect2i &roi_current)

I guess if you put more thought into it, you could come up with a more CPU friendly strategy.
GDAL warper is ridiculously slow

OpenREALM/modules/realm_ortho/src/gdal_warper.cpp

Line 169 in 465ecc8

warping.ChunkAndWarpImage(0, 0, GDALGetRasterXSize(dataset_mem_dst), GDALGetRasterYSize(dataset_mem_dst));

The transformation into EPSG:3857 seemed to me like a shift and maybe a slight rotation, nothing too expensive. But it takes more than a second to compute which is probably around 90%< of the processing time the tileing requires.

- TMS is a standard that isn't used by a lot of mapping software, so allow usage of non-tms tiles (Google/OSM) as well with a config flag.

* -1 can be used in max zoom level to select zoom based on GSD * Max zoom can be overidden if it is significantly higher than the available GSD to avoid oversampling an image.

- To help save/load cache files more easily, the stage can be initialized with a specific path for all cache items, rather than the default stage path. - If a path is not specified, operation will be unchanged.

- Shutdown is cleaner if we join the cache thread when the other threads shut down. Otherwise, the cache thread may keep running up until the program exits, causing odd behavior.

- To allow the system to reset after subsequent runs, it can be useful to have the system automatically delete cache tiles when starting a new run. - This is added as a command line option to allow enabling/disabling of the behavior - This is preparation for allowing a job to resume (sans old keypoints) after a crash

- This was causing crashes on iOS for a currently unknown reason. It was removed since the file I/O isn't expected to be called while the application is running anyway.

- To better integrate with external systems, a message that occurs when cache files are written to disk to be consumed by other applications is added. - Contains: - Path to the tile directory w/ x/y/z tiles - The type of tile (png) - The region of tiles updated at each zoom level - The topic - Partial updates and full updates are both written out when new tiles are written to the drive - Required moving tile_cache into the tiling stage for tighter integration with messages [ES-217]

- Added new parameter to tiling.yaml to control the option - All tiles are placed into the cache structure, but no data is loaded by default - Right now, it expects to find color_rgb, elevation_angle, and elevation data to work properly [ES-218]

- The previous roi request wasn't being saved, causing the prediction to always just use the current roi. Fixed. [ES-218]

- When loading tiles from the disk, you wouldn't get an update that new tiles were present when loading from the cache. This update publishes the topic a single time with the loaded tiles when the tiling node is first started. - We wait until the node is started to ensure all listeners are set up first. [ES-218]

zthorson added 3 commits August 3, 2021 12:53

Handle case where roi of tiles doesn't fit in the new merged matrix

70c0d4a

Fix NaN comparisons that may not be valid w/ OpenCV

6d627fa

- Comparisons using OpenCV and NaN numbers can have undetermined results due to some shortcuts OpenCV takes in performing comparison operations - This can results in odd behavior where only some of the NaNs are included

Update blend in tiling to match mosaicing method

76c5bfd

zthorson marked this pull request as draft August 3, 2021 18:11

zthorson force-pushed the upstream/tiling branch from 70b71e0 to 9829386 Compare August 5, 2021 21:58

zthorson added 5 commits August 11, 2021 12:40

Allow selection of TMS or Google/OSM Tiles

161620e

- TMS is a standard that isn't used by a lot of mapping software, so allow usage of non-tms tiles (Google/OSM) as well with a config flag.

Add settings for min/max zoom levels

6caf641

* -1 can be used in max zoom level to select zoom based on GSD * Max zoom can be overidden if it is significantly higher than the available GSD to avoid oversampling an image.

Allow specifying the tile cache location directly

08b7fb2

- To help save/load cache files more easily, the stage can be initialized with a specific path for all cache items, rather than the default stage path. - If a path is not specified, operation will be unchanged.

Move thread join to finish callback for shutdown

cf6a492

- Shutdown is cleaner if we join the cache thread when the other threads shut down. Otherwise, the cache thread may keep running up until the program exits, causing odd behavior.

zthorson force-pushed the upstream/tiling branch from 062a04d to 00d50fc Compare August 17, 2021 19:46

zthorson added 5 commits August 24, 2021 16:10

Remove uneeded mutex on file i/o

a79df17

- This was causing crashes on iOS for a currently unknown reason. It was removed since the file I/O isn't expected to be called while the application is running anyway.

Load tiles into tiling cache on init

844a7bf

- Added new parameter to tiling.yaml to control the option - All tiles are placed into the cache structure, but no data is loaded by default - Right now, it expects to find color_rgb, elevation_angle, and elevation data to work properly [ES-218]

Fix cache prediction previous request not saving

3456423

- The previous roi request wasn't being saved, causing the prediction to always just use the current roi. Fixed. [ES-218]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DRAFT: Tiling Stage Fixes and Improvements #65

DRAFT: Tiling Stage Fixes and Improvements #65

zthorson commented Aug 3, 2021 •

edited

Loading

laxnpander commented Aug 4, 2021 •

edited

Loading

zthorson commented Aug 5, 2021

laxnpander commented Aug 6, 2021

DRAFT: Tiling Stage Fixes and Improvements #65

Are you sure you want to change the base?

DRAFT: Tiling Stage Fixes and Improvements #65

Conversation

zthorson commented Aug 3, 2021 • edited Loading

Description

Reason

Method / Design

Testing

Other Notes

laxnpander commented Aug 4, 2021 • edited Loading

zthorson commented Aug 5, 2021

laxnpander commented Aug 6, 2021

zthorson commented Aug 3, 2021 •

edited

Loading

laxnpander commented Aug 4, 2021 •

edited

Loading